Field of the Invention
The present invention relates to a technique for displaying reconstructable image data.
Description of the Related Art
Until now, in the case where an image is captured with wrong focus adjustment of a camera, it has been necessary to redo focus adjustment and recapture the image. Also, in the case where an image including a plurality of subjects having different depths is obtained with each of the subjects in focus, it has been necessary to perform capturing with each subject in focus multiple times and composite a plurality of the obtained images. However, in recent years, a technique known as light field photography, according to which it is possible to acquire an image from multiple view points by adding a new optical element to the optical system, and later adjust a focus position (refocus) by image processing, has been developed.
Focus adjustment can be performed after capturing with the technique, and therefore the technique is advantageous in that failure in focus adjustment at the time of capturing can be corrected with image processing. Furthermore, the technique is advantageous in that a plurality of images in which any subject in the image is in focus can be obtained from one captured image by changing an image processing method, which reduces the number of instances of capturing.
In light field photography, directions and intensities (light field) of rays passing through a plurality of positions in a space are calculated from data obtained by capturing images from multiple view points. An image obtained in the case of being formed on a virtual sensor after passing through a virtual optical system is calculated using information on the obtained light field. By setting this kind of virtual optical system and sensor as appropriate, the above-described refocus is also possible.
As an image capturing apparatus for acquiring the light field, a Plenoptic camera in which a microlens array is placed behind a main lens and a camera array in which small cameras are arranged are known. With both image capturing apparatuses, multi-viewpoint image data obtained by capturing images of subjects in different directions can be obtained through one capture. That is, it can also be said that the light field photography is calculating an image to be acquired by a virtual sensor from multi-viewpoint image data under virtual optical conditions. Note that processing for calculating an image acquired by the virtual sensor is referred to as “reconstruction processing”, and in “reconstruction processing”, processing for changing a focus position is referred to as “refocus processing” in particular, hereinafter. As the refocus processing, a method of performing projective transformation on the acquired multi-viewpoint image data, on the virtual sensor, adding the resulting data together, and averaging the added data is known (see Japanese Patent Laid-Open No. 2014-158258).
Methods disclosed in Japanese Patent Laid-Open No. 2014-158258 and US Patent Application Publication No. 2008-0131019, for example, are examples of a method for displaying an image (refocus image hereinafter) that has been subjected to refocus processing while a focus position is changed. The method of Japanese Patent Laid-Open No. 2014-158258 discloses a method of showing a user whether or not a subject is included in a range in which the focus position can be adjusted, on a screen on which the refocus image is displayed. Also, the method of US Patent Application Publication No. 2008-0131019 discloses a method in which a user specifies a subject to be in focus through a screen on which a refocus image is displayed, and a refocus image in which the subject is in focus is displayed on the screen.
However, with the above-described conventional techniques disclosed in Japanese Patent Laid-Open No. 2014-158258 and US Patent Application Publication No. 2008-0131019, it is not possible to know whether or not a subject is a refocusable subject unless the user specifies the subject to be in focus. When an image is displayed in a camera, there are various types of data such as non-refocusable still images, refocusable still images and moving images, and other data. In particular, a refocusable still image and a non-refocusable still image cannot be distinguished from each other because these still images are displayed similarly as still images.